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paper outlines issues encountered and further work to be done in undertaking evaluation of such a 
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This paper reports on evaluation of a scheme to 
improve University teaching through action research 
over a five-year period, in the science, engineering, and 
technology division of a large Australian dual sector 
University. Between 2002 to 2006 this scheme directly 
committed approximately A$2 10,000 in grants and 
involved over 130 teaching and other staff in 
sponsoring projects of up to eight months’ duration, 
with a total of 34 projects completed. Evaluation was 
informed by the desire of the academic developers 
concerned with the scheme to engage more widely with 
staff in predominantly empirical disciplinary cultures, 
to be more accountable within a University business 
management paradigm, and to contribute to the 
scholarship of academic development. The paper 
provides evidence - in terms of quality, effectiveness, 
practicality, participation, and satisfaction - to show 
how this scheme enhanced the scholarship of teaching 
and learning in the University. The paper outlines 
issues encountered and further work to be done in 
undertaking evaluation of such a scheme. 

Description of the Scheme 

Action Research in Teaching and Learning 
(ARTL) was an initiative to enhance the quality of 
teaching and learning at a large Australian dual sector 
University by supporting staff to take an action research 
approach to improve some aspect of their teaching. 
ARTL was offered in one-third of the University, the 
Science, Engineering and Technology Portfolio (one of 
only three conglomerate faculties), which comprised 
ten Schools on two main campuses, had around 1,200 
full-time equivalent teaching staff, and represented one- 
third of the University’s technical/vocational (TAFE), 


undergraduate (UG), and postgraduate coursework (PG- 
C) teaching programs. It was piloted in the Life 
Sciences disciplines during 2002-2003, and then was 
scaled up across all of Science, Engineering, and 
Technology in 2004-2006. It was part of a suite of 
activities carried out by the Portfolio academic 
development group (or work unit) to implement the 
University’s teaching and learning strategy. It was 
offered to staff in only these disciplines because the 
Science, Engineering, and Technology academic 
development group operates within a de-centralized 
model to support only those staff. 

ARTL provided competitive small grants to staff 
within a structured framework of project management 
and professional development firmly focused on the 
scholarship of teaching and learning. Eligibility 
extended to all staff involved in teaching or supporting 
teaching, including staff in full-time, part-time, and 
casual positions on campus and in community or work- 
based settings. Key conditions of the guidelines for 
applicants are shown in Table 1. 

From 2002 to 2006 ARTL committed in total 
approximately A$2 10,000 in grants and involved over 
130 staff participants from every eligible Department or 
School, working with students in 35 different courses 
and programs of study. It sponsored five rounds of 
projects, each up to eight months in duration, with 34 
projects completed at the time of writing. Projects 
investigated topics such as student assessment, 
problem-based learning, graduate capabilities, and 
teaching and learning with technology. 

ARTL began as an initiative - modeled initially on 
a scheme previously conducted at another University as 
described in Radloff. de la Harpe and Wright (2000) - 
to build and strengthen a shared culture of reflective 
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TABLE 1 

Key Conditions of 2005 ARTL Guidelines for Applicants 


Projects are intended to allow staff to plan an improvement through change in their teaching practice, implement the change, collect data about 
the change, analyze the data, and report back in a scholarly manner. 


The objectives are to provide academic and teaching staff in the Portfolio with the opportunities to: 

• Research real-world issues and practices in their teaching to improve student learning and student satisfaction; 

• Learn to use or improve the use of action research methodology for this purpose; and 

• Build a research profile by preparing an article for publication in a refereed academic journal 
For the Portfolio and the University, the scheme aims to: 

• Provide a framework for continuing professional development in teaching; 

• Develop and foster the scholarship of teaching and learning; and 

• Support progress towards targets for teaching quality improvement. 

The scheme aims to enhance teaching and learning in the Portfolio by strengthening priority areas such as: 

• Good teaching, as reflected by indicators such as national surveys; 

• Student development of graduate capabilities; 

• Student progression and completion rates; and 

• Graduate employment / self-employment / enterprise formation. 


Project proposals should: 

• Be developed on the basis of sources of evidence such as formal student feedback or key performance; indicator data, or current issues 
in teaching & learning literature — and make these clear in the application 

• Address an important problem or issue where the solution will: 
o Benefit students in large classes or cohorts and/or 

o Be shared by more than one course and/or 
o Impact on more than one program; 

• Apply to teaching in a course conducted by staff employed by the Portfolio; 

• Achieve a result within eight months and an outcome into the future; 

• Propose a team rather than one member of staff — teams may be drawn from course or program teams, and may also include 
interdisciplinary and cross-sector members; and 

• Be consistent with work planning of the School/s where project participants are based. 


Proposed projects should focus on at least one of the following priorities: 

• Improving approaches to assessment; 

• Enhancing student leadership capabilities; 

• Strengthening work-integrated learning; 

• Internationalising the curriculum; 

• Innovating in teaching and learning with technologies; and 

• Strengthening student transition. 


and scholarly practice in teaching. Its approach was to 
offer professional development that recognized 
disciplinary cultures and ways of knowing, and that was 
situated in the everyday work of teaching. Its operation 
was resourced in three main ways: direct funding 
support for projects, staff time invested by project 
teams, and coordination and management overheads 
(half of the year-round workload of a full-time middle- 
level academic developer, under direction). 

Project funding amounts were modest by the 
standards of “big research” (from A$ 1,000 in 2002 up 
to A$5,000, A$8,000, A$ 10,000 per project in 2003, 
2004, 2005 respectively), and funding was disbursed 
against milestones committed to by successful 
applicants which included requirements to: attend a 
methodology workshop, (re)develop a project design, 
draft and submit an ethics application, conduct the 
action research project with students and the staff team. 


submit one-page written progress reports monthly, 
work collaboratively with a project mentor with at 
least monthly contact, present at two project progress 
seminars and the annual portfolio teaching and 
learning forum, and submit an article for publication 
and an evaluation of the project experience. 

Among the many schemes and many approaches 
to funding research into teaching in higher education, 
ARTL had a number of distinguishing features 
relating to the provision of structured end-to-end 
support and professional development for participants. 
This was intended to improve outcomes from the 
schemes in four ways: by fostering research into 
teaching among staff with little or no previous 
experience of it, by including targets and indicators 
for teaching and measurable scholarly outputs for 
research, by increasing access to professional learning 
and development on many levels in the University and 
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its community, and by securing the engagement of a 
wide range of stakeholders. 

An Action Research Approach 

Action research is well established as an approach 
to practitioner research, and is located in a 
phenomenological and interpretive paradigm (Zuber- 
Skerritt 2003). In ARTL, action research was selected 
to assist academic staff who typically were formally 
trained in natural and physical science research 
methods, but less experienced in methods suited to 
researching their teaching practice. Action research was 
selected as it is characterized by: investigating a 
complex, real-world question; reflecting on 
participants’ professional practice; using cycles of 
action and critical reflection; often using qualitative 
data, although quantitative data can be used; and 
developing a solution to a specific situation, rather than 
a causal explanation (Dick & Swepson 1997; Zuber- 
Skerritt 2003). A significant body of literature supports 
the use of action research in this context (see for 
example. Cherry 2002; Kember 1998, 2002). 

A Framework for Evaluation 

ARTL was designed to be conducted within a 
framework of continuous evaluation and improvement, 
so regular review of this scheme was an integral part of 
the reflective practice of the academic developers who 
sponsored it. But even though coordination and support 
provided through the scheme, as refined progressively 
over the five year period, seemed to be set at an 
effective level to achieve stated aims, continued 
operation of this scheme needed to be evaluated more 
formally. Academic development units are more than 
ever subject to the forces of restructuring and the 
demands to engage with a range of stakeholders, 
including client groups, senior administrators, and the 
profession (Gray & Radloff, 2005). 

Evaluation of ARTL sought to address the 
situation described by Walshe (2002), wherein several 
factors are slowing the development of new knowledge 
in the education sector: the opportunity to conduct 
educational experiments is limited, most of the practical 
knowledge remains tacit, and much of the innovation 
doesn't connect with formal research or get 
disseminated. Opportunities for staff to undertake 
research on teaching and learning are limited given the 
competitive nature of higher education and the 
emphasis on disciplinary research as a measure of 
institutional quality and international standing. In 
addition, success in research leads to peer recognition 
and academic advancement. There is thus both pressure 
and incentive for staff to engage in discipline -based 


research rather than in research into the pedagogy of 
their discipline. 

Further, given the traditional view of teaching as a 
private activity involving what Shulman (1993) calls 
“pedagogical solitude,” there is little opportunity or 
incentive for University teachers to share knowledge 
and experience gained through their educational 
practice. Such sharing is also limited by a lack a 
common language of teaching and learning. Moreover, 
many academics do not have experience in evaluating, 
documenting, and disseminating educational practice in 
ways that are perceived as scholarly and conforming to 
traditional research paradigms. 

The decision to evaluate ARTL in depth was 
informed by the desire of the academic development 
unit for stronger understanding among stakeholders in 
the academic disciplines, clearer accountability within 
the University’s business management framework, and 
more substantial contribution to the scholarship of 
academic development. 

However, an approach to evaluation was not 
straightforward. As McLoughlin and Samuels (2002) 
have outlined, underlying any evaluation of 
interventions to improve teaching are at least four 
educational discourses, or ways of framing the very 
meaning and purpose of improving teaching. One is a 
discourse about reflective practice, which involves 
critical examination of current practice, and becoming a 
reflective practitioner through self-evaluation of one’s 
teaching skills, attitudes, and conceptions of teaching 
and learning. Another is about the interdependence of 
teaching and (educational) research, such that research 
findings should inform and improve the practice of, and 
be meaningful and accessible to, practitioners whose 
main discipline is not education. Another focuses on 
inquiries into student learning, particularly influenced 
by phenomenographical research. Finally, there is the 
discourse about teaching as a form of scholarship that 
requires higher education teachers to be well informed 
about educational research, to be self-reflective and 
committed to improving student learning, and to be 
investigative and communicative about teaching. 

As well, as McAlpine and Harris (2002) have 
outlined, there are multiple forms of practice that need 
to be considered in evaluating teaching improvement, 
including subject matter expertise, design skills, 
delivery skills, management skills, skills in mentoring 
learners, personal professional development practices, 
and organizational development practices. The 
evaluation of ARTL sought to factor in ideas of worth 
that would accommodate a range of stakeholder 
perspectives and constructs of teaching practice. 

For such reasons, strict “value for money” or 
“return on investment” performance auditing - that is, 
evaluation in terms of economy (the acquisition of 
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resources on the best possible terms), efficiency (the 
use of resources to achieve a given level of output), 
and effectiveness (the match between intentions and 
outcomes) - seemed unsatisfactory for evaluating a 
scheme such as ARTL, for, as Elliott (2002) has 
noted, such an approach could have undesirable 
outcomes such as: possible erosion of academics’ 
motivation to innovate and of trust in academics’ 
capabilities; shifting of focus onto readily observable 
outputs, rather than on chain-of-effect outcomes; and 
failure to capture time-dependent and context-bound 
aspects of improvement. 

The framework ultimately chosen as most 
appropriate to analyze and report on academic 
developers’ observations about the value and worth of 
the ARTL scheme was adapted from the criteria 
established by the Australian Awards for University 
Teaching (AAUT) to recognize and reward excellence 
in institutional efforts to enhance the quality of 
teaching and learning (see The Carrick Institute, 
2005). These criteria framed the work and outcomes 
of the scheme in a way that would make it possible to 
capture the various discourses, multiple practices, and 
organizational performance issues within the broader 
work of enhancing the quality of teaching and 
learning; they were endorsed by a peak national body, 
the Australian Universities Teaching Committee, and 
thus were validated externally to the academic 
development unit and the University itself; and the 
ARTL scheme was eligible to be considered for 
recognition within them. Specifically, the AAUT 
criteria address the extent to which an institutional 
project or initiative practices the following: 

• facilitates quality practice; 

• assists staff to be more effective; 

• is innovative and practical; 

• achieves participation/penetration; and 

• achieves client satisfaction. 

The following five sections of this paper report on 
evaluation carried out by the academic developers 
concerned, using the AAUT criteria to conduct a 
systematic longitudinal review of documented ARTL 
operation. 

Extent to Which ARTL Lacilitated Quality Practice 

Kember (2002) describes the dilemma in 
facilitating an action research scheme: 

There was a desire for the project teams to retain 

ownership of their projects and to conduct all 

aspects themselves. The participants would 


therefore be motivated and would learn from the 
experience. At the same time, though, sufficient 
assistance, advice and support needed to be 
provided to ensure that the projects were 
successfully conducted, and achieved their aims, 
(p. 89-90) 

A distinguishing feature of ARTL was its 
provision of structured end-to-end support and 
professional development for participants to facilitate 
quality outcomes from the scheme, illustrated here 
through its project management, mentoring, brokering, 
and advocacy work. 

Project Management 

ARTL took a strong project management 
approach to the organization, management and 
coordination of the scheme, with the coordinator 
undertaking the following tasks : 

• plan and promote annual round and take 
inquiries; 

• advise on development of applications and 
receive applications; 

• coordinate selection process including panel; 

• coordinate expedited ethics application 
process; 

• monitor monthly team progress reports and 
budgets; 

• process casual staff employment and other 
accounts; 

• run methodology workshop and progress 
seminar for teams; 

• induct and support mentors; 

• events-manage day-long end-of-year forum 
based on project presentations; 

• support guest speakers and panelists to 
present at workshops and forums; and 

• produce monthly management reports. 

Mentoring 

ARTL introduced a formal mentoring system for 
project teams. Mentors were staff experienced in 
educational research or educational development, but 
removed from the administration of the scheme and 
with no authorship role in writing up the project; they 
acted as critical friends to the project team. They 
engaged in such activities as coaching the project team 
in effective teaching and learning interventions, 
helping the team to reflect on their experiences, and 
undertaking preliminary review of planned 
presentations or publications about the projects. 
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Brokering 

ARTL facilitated the work of teams by brokering 
assistance from third party support service providers 
across the University as the need arose, for example 
by the following actions: negotiating appropriate 
media production services for research into the use 
of virtual field trips, sourcing IT services technical 
solutions to support research into student e- 
portfolios, and mediating where misunderstandings 
arose during third party support. 

Advocacy 

ARTL undertook advocacy on behalf of teams 
regarding University policies and procedures in 
sometimes unexpected areas; for example, ARTL 
assisted in streamlining and interpreting the human 
ethics approval processes to capture the 
circumstances of practitioner research into teaching, 
and it assisted in clarifying processes for recruitment 
and appointment of casual research assistants in 
research-into-teaching settings. 

Extent to Which ARTL Assisted Staff 
to be More Effective 

ARTL specifically aimed to strengthen a culture 
of the scholarship or research into teaching. Diamond 
(2002) highlights competing notions of scholarship, 
arguing: 

Lor the greater part of the twentieth century, 
most professors paid little attention to defining 
the term “scholarship” or to addressing what was 
meant by “scholarly work.” Most individuals and 
disciplines bought into the concept growing out 
of the sciences that to be scholarly an activity 
needed to be “original” research that led to 
publication as a book or an article in one of the 


more significant discipline-based, refereed 
journals. . ..The impact of this approach, while 
seldom mentioned publicly, was at times 
extremely unfortunate not only for individual 
faculty members but also for the disciplines 
themselves, (p. 73) 

ARTL fostered research into teaching among staff 
who may have had little or no previous experience of 
educational research, or of a team-based or action- 
oriented approach to educational research, in several 
key respects. 

Shifting Research Paradigms 

Staff in science, engineering, and technology 
disciplines more than often have developed their 
academic practice within a quantitative or empirical 
research paradigm. In contrast to this positivist 
research tradition, action research is typically quasi- 
experimental, usually uses qualitative methodologies, 
is conducted in naturalistic settings, and does not 
usually provide causal explanations, but rather is 
focused on obtaining valid data that is “rich, real and 
deep” (Dick & Swepson, 1997). Lurther, the notion 
of reflecting on practice is not generally well 
understood or routinely applied in higher education 
(Davis, 2003). To pursue change and knowledge 
together through action and reflection on the action 
was, therefore, a new way of working as researchers 
for most staff who participated in ARTL. 

Inclusiveness 

ARTL contributed to the professional 

development of diverse staff, as shown in Table 2, 
including early and mid-career academics, and also 
groups for whom continuing professional 

development in teaching is not always accessible: 
general staff, sessional staff, and graduate students. 


TABLE 2 

Analysis of Participating Staff in ARTL 2002-2005 



Professor 

Associate 

Professor 

Holding PhD 

Higher Ed 
academics not 
fitting into 
previous 
categories 

Technical and 
Lurther 
Education 
(TALE) 
teaching staff 

General staff 

2002 

0 

2 

8 

6 

1 

1 

2003 

0 

5 

3 

6 

0 

0 

2004 

0 

3 

17 

7 

7 

8 

2005 

3 

4 

12 

11 

3 

2 

Total 

3 

14 

40 

30 

11 

11 


Note. General staff includes educational designers, library staff, IT support staff, and student support staff 
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Legitimating Research into Teaching and Learning 

ARTL provided staff with a way to be formally 
recognized for their classroom research. It was used by 
participants in building teaching portfolios, applying for 
academic promotion, and nominating for teaching 
awards. The trend in 2005 for more participation by 
staff at professorial level gave added weight and 
prestige to involvement with the scheme generally. As 
well, participation by senior academics improved the 
opportunities for peer learning in ARTL workshop 
settings where these staff contributed their more 
extensive experiences as teachers and researchers. 

Conversing Across Disciplines 

The science, engineering, and technology division 
of the University comprised a range of disciplines - 
from aerospace engineering through cartography and 
complementary medicine to foundation studies - across 
which academic and professional discourse often 
struggled. Through the ARTL scheme, project teams 
who were workshopping applications, testing new 
practices, and interpreting student feedback found 
common cause and shared learning about teaching in a 
spirit not seen in many other University settings. 

In practice, the opportunities for conversing across 
the disciplines arose in workshop situations - 
specifically designed along an adult learning model in 
which participants were encouraged to support each 
other by sharing their understandings and professional 
expertise - and from 2005 forward, through an online 
discussion forum for ARTL project teams. This resulted 
in a community of practice, with projects enriched by 
professorial level staff from other projects sharing 
literature and insights, and mid-career staff who were 
more familiar with action research approaches having 
input into others’ projects. 

Extent to Which ARTL was Innovative and Practical 

ARTL contributed to a range of outcomes for an 
academic development unit that strove to operate within 
a framework of strong accountability and transparency 
to accomplish the following: 

• achieve institutional missions; 

• implement institutional strategic plans; 

• enhance student and staff experiences; 

• improve overall student and staff satisfaction; 
and 

• enhance student learning and research 

outcomes (see Blackmore et al., 2004). 

ARTL brought together within sponsored projects both 
strategic targets and indicators for teaching 


performance, as well as measurable scholarly outputs 
for research. It heightened awareness of the ethics of 
teaching, and it delivered economies of scale for 
supporting research into teaching. 

Strategic Focus 

The design of ARTL reflected University 
performance targets and high-level indicators for 
teaching and for research. Guidelines for projects (as 
shown in Table 1), including priority areas for action 
research, were cross-referenced closely to the detailed 
objectives in the University’s teaching and learning 
strategy. 

Scholarly Outputs 

ARTL participation committed staff to engage 
actively with the educational research community 
through reporting on their projects in refereed forums. 
Participants were supported to review relevant 
literature, draft publishable papers, and identify 
appropriate disciplinary or generic educational 
presentation and publication opportunities. National and 
international dissemination of ARTL projects is known 
to have occurred in the form of conference 
presentations or journal articles - at least seven in 2002, 
six in 2003 and eight in 2005. 

Ethical Stance to Working with Students 

The coordination of ARTL projects in 2002-2003 
identified scope for improvement in ethical practices 
around research into teaching, in particular regarding 
efficacy and protection of student learning (balancing 
the desire to improve teaching practice with the risk of 
adversely affecting student learning), informed consent 
and voluntary participation, vulnerability and unequal 
power relationships, and intellectual property and 
collegiality. As a result, all 2004-2006 participants were 
briefed on ethical issues in working with students, and 
they sought and obtained formal human ethics approval 
for their projects, as described in more detail in Chang, 
Gray, Polus and Radloff (2005). 

Long lead times and a lengthy application form 
based on bio-medical research were identified as 
potential barriers to ARTL participants applying for 
human ethics approval. This led to academic 
development group staff working collaboratively with 
the human ethics committee to introduce an expedited 
human ethics approval process for ARTL projects from 
2005 and to pilot a simplified 5 -page ethics application 
form for research into teaching practice in 2006. These 
practical innovations have had influence beyond the 
ARTL scheme; for example, the simplified application 
form will be rolled out across the University. 
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Econom ies of Scale 

Finding the time to do research into teaching is a 
pressing issue for most academic staff. Project teams 
typically used their ARTL funding for employing a 
research assistant, paying for efficient ways to collect 
and analyze data, and employing sessional staff to 
allow partial time release from teaching. However, all 
such expenditure was disbursed, and all casual staff 
were employed, by the ARTL coordinator on behalf of 
project teams in order to reduce the administrative 
overheads of research into teaching and thus to enable 
teaching staff to focus on core aspects of their projects. 
This approach to administration of the projects 
maximized the opportunity for a large number of staff 
to spend time on non-trivial professional learning and 
development activities. 

Extent of ARTL Participation and Penetration 

Participants in action learning typically experience 
a range of benefits including, as Bourner, Cooper and 
France (2000) enumerate, their own personal think- 
tank, a sounding board for testing out their ideas, 
traction and motivation to make progress, set-aside time 
and space for reflection, vicarious learning, active 
learning, and self-help group work. Beaty (2003) 
observes, “[Bjecause it is project focused, action 
learning can have wider benefits to the University as 
well as personal benefits for staff’ (p. 16). Taking 
account of all these factors, ARTL made professional 
learning and development accessible on many levels 
including to project staff teams and their students, to 
wider staff audiences in University operating units and 
at University forums, and to the wider community 
through public dissemination. 

Direct Involvement of Staff 

Applying for an ARTL project attracted increasing 
interest from staff each year, as shown in Table 3. 
Improvements to the application process in 2005 
obviated the need for full applications through an 


improved system for handling informal enquiries. 
Unsuccessful applicants received feedback on their 
applications and advice on other potential avenues to 
seek support. Over the five years of ARTL operation, 
successful projects were sited in every eligible 
Department / School. 

Dissemination Within the University and Beyond 

ARTL and project outcomes have been 
disseminated in a number of ways. ARTL culminated in 
a major teaching and learning forum at the end of each 
year of operation. Attendance at each of the 2002 and 
2003 forums was over 50, and in 2004 and 2005 it was 
over 100. Selected projects were the focus of 
presentations to staff at whole-of-University events 
such as teaching and learning seminars and research 
seminars. ARTL coordinating staff presented papers 
about aspects of the project to national or international 
academic development conferences in 2003, 2004 and 
2005. In addition, a number of papers have been 
published. Finally, an ARTL website was made 
publicly accessible at http://www.rmit.edu.au/set/ 
ad/sotl/artl. 

Other Organizational Learning and Development 

ARTL contributed to learning and development in the 
University, over and above that of project teams and 
their students, in a number of other ways. Project 
mentoring introduced formally in 2004-2005 gave at 
least 10 non-project-team staff the opportunity to 
develop and refine mentoring skills. Further, working 
partnerships and relationships with teaching staff were 
strengthened in response to the needs of the scheme, 
including among University curriculum developers, 
librarians, field or clinical supervisors, and employer 
advisory groups. The scheme also led to wider 
adoption and application of some innovations in 
practice, for example, student e-portfolios, internet 
videoconferencing, peer tutoring, and electronic 
journal clubs were piloted in ARTL projects and 
subsequently taken up in other courses and programs. 


TABLE 3 

Direct Interest and Participation in ARTL 2002-2005 

Expressions of interest Successful teams 

2002 8 applications 18 staff from 4/5 Departments / Schools 

2003 15 applications 14 staff from 4/5 Departments / Schools 

2004 29 applications 42 staff from 10/10 Schools 

2005 50 enquiries and 14 applications 35 staff from 9/10 Schools 
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Degree of Client Satisfaction with ARTL 

ARTL showed evidence of addressing and meeting 
the needs of different stakeholders in the scholarship of 
teaching, that is, not only project participants, but also 
students and University management. Westin and 
McAlpine (2001) propose that there is “a continuum of 
growth toward the scholarship of teaching that 
highlights the possibility of growth or development 
within and across three phases... growth in own 
teaching, dialogue with colleagues about teaching and 
learning, and growth in scholarship of teaching” (p. 96- 
97). ARTL project participants repotted satisfaction 
that reflects these three phases of growth. A sample of 
feedback from project participants is given here: 

Impact on Staff 

Twelve participants who did a formal evaluation of 
the 2002-2003 ARTL schemes (reported in Jansz-Senn, 
Chang, Gray, De Pew & Radloff, 2003) considered that 
the scheme had improved: 

• the quality of teaching and learning in more 
than one program (7/12); 

• their understanding and undertaking of 
collaborations to improve teaching (8/12); 

• their knowledge and / or practices of assessing 
students (9/12); and 

• their ability to be more reflective in their 
teaching practice (10/12). 

Comments from 2004 participants include: 

• “I experienced much value and benefit simply 
by completing the ARTL application process.” 
(team leader) 

• “My discussions with [the ARTL coordinator] 
were very helpful, and I found her very 
accommodating to assist in whatever way was 
possible. Also some good suggestions from 
her about how to proceed with some aspects of 
the proposal that I was concerned about.” 
(applicant) 

• “You are to be congratulated on the way the 
afternoon was organised and conducted... I 
learnt a great deal as a result of the workshop 
particularly around the issue of ethics.” 
(mentor) 

Impact on Students 

ARTL project reports conveyed an array of insights by 
staff into student learning and student satisfaction as the 
following comments illustrate: 


• “It was also found to be important to 
contextualize the e-portfolios within the 
overall program experienced by the student, 
and it was apparent that effective team 
teaching is necessary in order that students 
gain the full benefit of their portfolio creation 
skills in subsequent years of their program." 
(Allan, Zylinski, Temple, Hislop, & Gray, 
2003, p.579) 

• “preliminary studies into the effectiveness of a 
teaching method designed to encourage 
students to pose their own questions as an 
assignment task. . .provide considerable 
feedback on the progress and / or engagement 
of students with the material.” (Merchant & 
McGregor, 2004, p.l) 

• “focus group comments showed a wide 
variety of opinion about the two methods, 
suggesting that any one method of teaching 
will put some students at a disadvantage. This 
indicates that a varied approach to . . . teaching 
is desirable because it is more likely to catch 
the interest and attention of a wider number of 
students.” (Henry, Salter, Quazi, Bezen, Llynn 
& Kaul, 2004, p.l) 

Institutional Recognition 

The ARTL scheme was unique and much admired 
within the University. It received a University award in 
2005 for its innovative and practical approach to the 
enhancement of the quality of teaching and learning, 
and it was nominated to represent the University in the 
national Australian Awards for University Teaching 
competition in this category. 

further Considerations 

findings about ARTL in relation to the five criteria 
above may offer lessons for good practice in supporting 
the scholarship of teaching. They also represent a 
significant effort to use evidence to assert the worth and 
value of this type of academic development work. 
However, in this respect there is still more work to be 
done, in order to make explicit what didn't work or 
couldn't be shown about the scheme, triangulate the 
evaluation done by scheme participants and 
administrators, and drive further evaluation and proper 
follow-up action. 

Unknowns 

One of the least successful aspects of ARTL was 
that, in a dual-sector setting where greater mutual 
exchange between technical/vocational (TALE) 
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educators and higher educators was an institutional 
goal, TAFE staff participation in ARTL was 
proportionately less, and less successful, than higher 
education staff participation. It is not clear why this was 
the case, but it is possible that this is a reflection of 
differences in the staff development and research 
contexts of the two sectors (which have separate 
performance measures and career structures overseen 
by different levels of government) that the ARTL 
scheme alone could not overcome. 

One of the great unanswerable issues about ARTL 
concerned the opportunity cost of participation in it that 
was borne by participating staff. Especially in the eyes 
of teaching staff and their academic managers, ARTL 
often seemed to compete for scarce time and attention 
with other priorities such as discipline-based research, 
academic administration, and curriculum renewal, as 
well as other staff development needs and options. The 
actual time commitment of a participating staff member 
was “as long as a piece of string,” variable from one 
project and one team member to another, and not 
feasible to calculate as total person-hours. 

For the academic developers involved in ARTL, 
evaluation raised a long line of questioning about what 
is proper professional conduct in terms of academic 
developers’ affiliations with staff action research 
projects, e.g., in what circumstances could one be on 
the selection panel, could one be a team member, could 
one say that some of the ARTL data belonged to him or 
her? Some of the issues of academic integrity, 
intellectual property and research ethics are detailed in 
Chang, Gray, Polus, and Radloff (2005). Questions also 
persisted about what might be the best location within 
the University for the sponsorship of such a scheme in 
order to optimize engagement by “academic tribes” at 
the same time as reaping cross-disciplinary benefits. 

Triangulation 

ARTL was designed to be conducted within a 
framework of continuous evaluation and improvement. 
Triangulation provides data from multiple sources that 
can be used for evaluation purposes. In the case of 
ARTL, to date the sources of data used to evaluate the 
project have included the academic developers who 
designed and implemented ARTL and other staff 
involved in implementing ARTL, the staff who 
undertook ARTL projects and, in some instances, their 
students who were participants in projects, as well peers 
who contributed to ARTL projects in different ways, 
including as mentors and reviewers. It is possible to 
understand and further strengthen the evaluation of 
ARTL by using Kember’s (1998) model for 
triangulated evaluation of an action learning project in a 
tertiary setting, which maps onto Guskey’s (1999) 
taxonomy for evaluating professional development. In 


Kember’s (1998) model, academic developers design 
and monitor the overall scheme (planning evaluation), 
teaching teams reflect on and report on progress within 
their own projects (formative evaluation), and 
independent formal evaluation is commissioned 
(summative evaluation). 

Specifically in ARTL, planning evaluation can be 
understood to have adequately occurred through the 
way that the ARTL coordinator captured and worked 
with a variety of data such as progressive feedback and 
debriefings by applicants (successful or not), workshop 
participants, mentors; project staging and budget 
reports reviewed by Portfolio managers; broader staff 
participation rates and comments from learning and 
teaching forums annually; and referees’ and colleagues’ 
external reviews of written and presented accounts of 
the scheme. Formative evaluation can be understood to 
have adequately occurred through the way that each 
project team captured and worked with evaluative data 
for their own project by meeting monthly with a 
mentor, maintaining monthly written progress reports, 
reporting at the progress seminars for all teams, 
gathering students’ and other teachers’ feedback on 
their project, and inviting referees’ and external 
colleagues’ peer review of their formal project reports. 
Summative evaluation can be understood to have 
occurred only partially to date through the evaluation 
reported in the present paper by members of the 
academic development unit that sponsored the scheme. 
Accordingly, planning has been done for an evaluation 
of the 2002 through 2005 operation of the scheme by an 
evaluator external to the University. A survey 
instrument has been designed for use with key 
stakeholder groups to evaluate the value and worth of 
ARTL in terms of Guskey’s (1999) criteria for 
measuring effectiveness of professional development, 
and survey targets and criteria to be evaluated are: (a) 
project team leaders, other participating project staff, 
and mentors on their reaction to taking part; (b) 
coordinators of courses and programs and students in 
courses in programs in which projects occurred on their 
observations about learning and application arising 
from (a); (c) coordinators of courses and programs in 
which projects occurred and other academic managers 
of participating staff on their assessment of macro-level 
outcomes for teaching arising from (a) and (b); and (d) 
all of the above on their sense of organizational 
learning and development arising from (a), (b) and (c). 

Moving Further 

While recognizing the value of further, external 
evaluation, it is important not to downplay the 
information and knowledge management achievements 
that underpin the present paper, nor to overlook them as 
an essential precursor to inviting in an external 
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evaluator. The operation and management of ARTL 
between 2002 and 2005 was difficult to routinize 
because it happened during a period of prolonged and 
extensive organizational change. There was no 
certainty of continuation for the scheme from one 
year’s University budget to the next in any year of its 
operation. Various approaches to allocating the 
coordination workload were tested over the four years 
of operation, and the role was not assigned to a 
permanent member of staff until mid-2004. Those 
who would undertake the scholarship of academic 
development are often challenged by such churn in 
their organizations and across the higher education 
sector. 

Specific actions that would assure continued 
improvement of ARTL into the future include: 
carrying out external, independent evaluation of the 
scheme and adopting ensuing recommendations; 
establishing an ongoing budget line item and base 
funding; capturing the intellectual property of the 
ARTL coordinator in a suitable form (such as a 
handbook or resource kit) to ensure internal 
information management and to share with academic 
developers elsewhere; and rolling out ARTL across 
the rest of the University, and potentially across other 
universities in a coordinated fashion as a scholarly 
initiative. 

Based on our experiences, we believe that it is 
worth persisting. Only in this way will all those with a 
commitment to improve teaching and its scholarship 
be able to make evidence-based decisions, knowing 
whether or not schemes like ARTL can provide the 
systems and structures to sustain project impacts, 
value teaching, and build the organizational culture 
that is needed for continuous quality improvement in a 
performance-oriented external operating environment. 
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